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Abstract 

This paper describes the framework for the development of OER-based learning materials TCC121/05 
Programming Fundamentals with Java for ODL learners in Wawasan Open University (WOU) using three 
main development phases mainly: creation, evaluation and production phases. The proposed framework has 
further been tested on ODL learners to promote greater use of OER repositories and further the acceptance 
of a wider range of learning resources. By using collective feedback sessions, the walkthrough of available 
OER resources to be integrated or assembled into the learning materials is illustrated in the course develop¬ 
ment cycle and the interaction among students and instructors and learning experiences of courses units’ 
content are evaluated at the end of each of the courses with the production of external course assessor’s 
reports (Quality Assurance). The learners’ interactions with OER materials in LMS have demonstrated the 
development of a model for effective knowledge transfer using OER. 

Keywords: collective feedback sessions; course development; learning materials; OER development; quality 
assurance 


Introduction 

Open Educational Resources (OER) include full courses, course materials, modules, textbooks, 
streaming videos, tests or other tools that resides in the public domain with intellectual property 
license that permits free-use or repurpose by using attribution, no derivatives, share-alike, non¬ 
commercial or any combinations of these CC licenses to support access to knowledge (Atkins, 
Brown & Hammond, 2007). The development of digital materials movement has encouraged users 
to re-use, revise, remix and redistribute resources (Hilton III, Wiley, Stein & Johnson, 2010) through 
appropriate tools and made available through creative common licensing. 

In preparation of the OER-based E-Learning course, workshops and seminars are held to develop 
capacity of participants for integration of OER in their own teaching and learning at Wawasan Open 
University (WOU). The OER-Asia initiative in 2010 under the auspices of the University for promot¬ 
ing and transforming the existing course development process is aim to increase the quality and 
efficiency of developing OER-based course materials (Menon & AN, 2012). The OER Policy for WOU 
with implementation strategies are given as follows: 

“WOU will promote and implement the creation, reuse, remix, repurpose and redistribution of Open 

Educational Resources (OER) within an Open Licensing framework” 

The OER repositories are introduced to the course team members in development of the 
OER-based course for the initial development by Chung and Khor (2012), TCC121/05 Programming 
Fundamentals with Java who comprised of Course Team Coordinator, Academic Members, Course 
Writers, Instructional Designers, Editor and External Course Assessor (ECA) for searching, creating 
and customizing learning contents including content modules, articles, books and journals. 
TCC121/05 was scheduled for course development in July 2012 as an OER integrated course 
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adopting the WOU-Open License Policy (OLP) (Menon & AN, 2012) and first presented in January 
2013 semester. The course development for TCC121/05 has been conducted smoothly within six 
months duration and the course materials were made available to the students at its six regional 
learning centers via WawasanLearn —a LMS based on the open-source system called Moodle. With 
the experience and feedbacks acquired from four semesters’ presentations, the development of 
TCC121/05 is conducted based on the students’ and tutors’ experience that have gone through the 
course. The course module together and OER contents provided in WawasanLearn formed an 
adequate support for students’ learning. 

According to McGreal et at. (2013), sharing OER can be a valuable way to disseminate knowledge, 
diversify teaching activities and gain new insights into other teaching methods for a particular subject. 
A number of technical issues relating to improving accessibility and usefulness of OER such as the 
use of open source software, increasing interoperability by using open standards and emerging 
technologies that affect the open educational resources movement are being investigated (OECD, 
2007). Motivations for learners to participate in OER development are varied, and many struggle 
to engage with courses and keep motivated in the context of an online learning environment 
(Yuan & Powell, 2010). In order to overcome the aforementioned issues, the Collective Feedback 
Sessions are initiated through live virtual domain using WizIQ to play the important role in helping 
to increase and widen participation in accessing and reviewing OER repositories. Virtual sessions 
were initiated to improve and overcome issue of how to find the resources that are most relevant, 
adding enriched metadata to resources for classification of educational resources (MELT, 2010). 

This paper presents the framework for the development of OER-based learning materials and the 
evaluation of the student achievement and the accessibility of the OER content in WawasanLearn. 
The first section overviews the course development phases including creation, evaluation and 
production phases. The next section describes the Quality Assurance (QA) implementation for 
the OER-based course development via Collective Feedback Sessions. The findings, surveys and 
evaluations are discussed in the third section. The final section describes the recommendation and 
conclusion. 

OER-Based Course Development Cycle 

Figure 1 depicts the development phases (creation -Revaluation -^production) of OER-based 
learning materials in this approach by the initiation of the creation of Course Syllabus, Course 
Development Timeline and Course Blue Print by Course Team Coordinator and respective course 
team members. In the creation phase, course units write up are produced and reviewed by the 
instructional designer before being sent to ECA to be assessed through evaluation of ECA (SOP 
for Course Development, 2010). It is important to ensure that QA standards were maintained in the 
development of the course materials. Thus, internal and external quality assurances are employed 
into the evaluation and production phases. 

As shown below, TCC121/05 comprises of 5 units course write up, self-tests, activities, examples 
and unit practice exercises which are written based on the integration and assembly of OERs under 
a CC BY-SA 3.0 license. 


Course Overview 


Introduction 


Unit 1 

• Unit 2 

o Unit content 

o Unit content 

o Self-tests 

o Self-test 

o Activities, examples 

o Activities, examples 
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Minutes Meeting: 

Minutes meeting are created and associated with each interim reports, course blue print, course guide, units write up to document the 
suggestions made with regards to the course team report. 

Figure 1: Development Phases of OER-based Learning Materials 


• Unit 3 

• Unit 4 

o Unit content 

o Unit content 

o Self-tests 

o Self-tests 

o Activities, examples 

o Activities, examples 


• Unit 5 

o Unit content 
o Self-tests 
o Activities, examples 

• Course Summary 

• Unit Practice Exercise 

The details of the framework documents are illustrated in Table 1. 

Quality Assurance via Collective Feedback Sessions 

The assurance is made aware throughout course development phases associated with the OER- 
based course materials with the inclusion of course writer’s guidelines, academic members’ inputs, 
ECA’s role and guidelines, course team reports, tutors’ and students’ feedbacks. TCC121/05 course 
team members are invited to attend Collective Feedback Sessions to ensure the review of each 
interim reports are taken place. Learners’ feedbacks and peer review processes are incorporated 
in the course development cycle as one of the most used quality assurance processes in academia 
(OECD, 2007). 

Learning environments that encourage active participation and dialogue session provide learners 
with opportunities to engage in a process of knowledge construction as they try to create meaning 
from new experiences (Jonassen, 1995). The discussions in the collective feedback sessions held 
during the course development focused on the “how to develop” and usage of OER particularly 
in Java programming related areas. Virtual Workshop Sessions engaged using the virtual domain 
via WizIQ are held with guidance and step-by-step OER creation sessions for the OER content, 
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Table 1: Production of Interim Reports and QA Framework for Course Development Cycle 


OER-based related production 
documents 

Course Development Team and QA Framework 

Course Syllabus, 

Course Development Timeline 

• Course Writers, Course Team Coordinator 

Course Blue Print 

• Minutes Meeting (Course Team Coordinator, Course Writers, 
Academic Members, Instructional Designers) 

• Course Team Report 

• ECA Report 

Course Guide 

• Minutes Meeting (Course Team Coordinator, Course Writers, 
Academic Members, Instructional Designers) 

• Course Team Report 

• ECA Report 

Course Units write up 

• Minutes Meeting (Course Team Coordinator, Course Writers, 
Academic Members, Instructional Designers, Graphics 
Designers, Learning and Library Services Members) 

• Course Team Report 

• ECA Report 

Assessments 

(Assignments, Examinations) 

• Minutes Meeting (Course Team Coordinator, Course Writers, 
Academic Members, Instructional Designers) 

• Course Team Report 

• ECA Report 

Completed Course Units 
(Recommendations, Actions Taken) 

• Final ECA Report (Final Evaluation Report) 


Table 2: Collective Feedback Sessions 


Collective Feedback Sessions 

OER Creation Initiative 
Discussion 

http://www.wiziq.com/online-class/ 

645825-creation-assurance-repurpose- 

and-sharing-of-oer 

Members attended: Coordinator, 
Course Writers, Library 
members 

OER Repositories 
Walkthrough 

http://www.wiziq.com/online-class/ 

644140-basic-guide-of-oer-and-oer- 

repositories 

Members attended: 

Coordinator, Academic members, 
Instructional Designers 

Assessment Discussion 
and Review of feedbacks 

http://www.wiziq.com/online-class/ 

636868-java-programming- 

assessment-discussion 

Members attended: 

Coordinator, Course Writers 

Exploring Open 

Educational Resources— 
Review of Assessments 

http://www.wiziq.com/online-class/ 

640403-review-of-assessments- 

ipohro-student 

Members attended: 

Coordinator, students 

OER Collective Feedback 
Session 

http://www.wiziq.com/online-class/ 

647666-oer-creation-cars-collective- 

feedback-session 

Members attended: 

Coordinator, Academic members, 
Library members, Instructional 
Designers, Students 
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Figure 2: Walkthrough of Module Creation in CNX 

(Module 1: image files, Module 2: Pseudocode for Java, Module 3: image files) 


intellectual property issues, resources that can be reuse, remix, repurpose and integrate can be 
accessed in the following sessions links as shown in Table 2. 

The study of adapting and remixing individual modules into courses (example illustrated using 
CNX) is presented in Figure 2. The creation of two modules illustrated in Figure 2 covers 
Module 1: Object and Classes (with image containing Symbols of Flowcharts), Module 2: Intro to 
Object and Classes (Pseudocode for Java) and Module 3: Java Programming (Control Structures, 
Arrays). The examples for the modules created consist of self-contained metadata which allows the 
users to choose varieties of language used and the subject categories. The creation module enable 
users to import CNXML documents (Words files, OpenOffice files, LaTex, multimedia assets) and 
select specific elements of the module to edit. The three sample modules created through the 
Collective Feedback Sessions are: 

• Module 1: Object and Classes http://cnx.org/content/m41591/latest/ 

• Module 2: Intro to Object and Classes http://cnx.org/content/m41398/latest/ 

• Module 3: Java Programming http://cnx.org/content/m41396/latest/ 

(Control Structures, Arrays) 


Findings and Discussion 

Key Findings: Summary of Key Information of TCC121/05 Programming Fundamentals with 
Java 

The summary of the key information of the students enrolled in undergraduate course TCC121/05 
Programming Fundamentals with Java is presented in Table 3 below: 

In quantifying the participation rate and total time of access, redundant data from the log files of 
the activities in the LMS have been eliminated to ensure a true reflection of the learners’ learning 
pattern with the online content. Table 3 shows the learners in OER-based TCC121/05 were higher 
in average age, with slightly more female learners and it was observed that there is an increase 
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Table 3: Summary of Key information of OER-based and Non OER-based course 



OER-based TCC121/05 
(January-July 2013) 

Non OER-based TCC121/05 
(August-December 2012) 

Number of learners enrolled for this 

course 

124 

129 

Average age 

32 

31 

Gender (Male/Female) 

49% Male, 51% Female 

48% Male, 52% Female 

Participation rate (%) 

(No. of Online / No. Enrolled) *100% 

98 

93 

Average frequency of activity over a 
semester 

(Total No. of Activity/No. of Learners 
Online) 

7.39 

6.2 


in the learners’ participation rate and higher average frequency of activity over a semester. The 
progress in the higher participation rate of learners in the LMS is also an evident to the wider access 
to OER learning. The exchange of learning questions and experiences in LMS help learners to 
clarify any doubt the have with regards to OER repositories and helping to find the most relevant 
and highest quality of resources. 

Key Findings: Patterns of Access in Online LMS WawasanLearn (views and posts) 

The pattern of access and activity by the course participants are observed in two separate semes¬ 
ters as depicted in Figure 3 {OER) and Figure 4 (Non OER). It was observed that the average 
activity level for OER-based TCC121/05 was higher than non OER-based TCC121/05 throughout 
the semester. The learners enrolled in OER-based TCC121/05 are found to be more initiated 
in accessing the LMS for OER online resources while non OER-based learners’ activity level is with 
a declining level towards the end of the semester. This indicates that the motivation for researching 



Month in the Semester 


Figure 3: Participants’ Activities (OER-based Course Material) 

(TCC121/05 Programming Fundamentals with Java: January - July 2013) 
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Figure 4: Participants’ Activities (Non OER-based Course Material) 

(TCC121/05 Programming Fundamentals with Java: August - December 2012) 


and utilising open content among OER-based learners are high as shown in the activity level of 
Figure 3. The students found that the WawasariLearn is useful as it supports the learning and they 
found that the forum discussion is relevant to the course contents using OER resources. 

Key Findings: Learners’ Interaction with OER Content and Resources in LMS 

The evaluation and feedback capability of the OER content and resources were investigated in 
terms of whether it permits one-way non-interactive or two-way interaction between learners—OER 
content to evaluate the outcome of learning arising from the interaction. Figure 5 shows the students 
were actively using the online discussion forums in WawasanLearn throughout the semester (58%) 
for knowledge-sharing activities. Learners in TCC121/05 used more than 50% of their time partici¬ 
pating in the online discussion forums that allow asynchronous exchanges of ideas among peers 
and with their instructors. This indicates that they value online resources which allow for two-way 
communication and also suggests that they are getting acquainted with active involvement in 
the online environment. It is also noted that the participants seek to acquire knowledge for self¬ 
enhancement and to apply the knowledge gained in their daily work. 

Learners in TCC121/05 spent more time in accessing the OER course content (19%) as the 
material involves providing constructive feedback and detailed commentary on course content via 
critical thinking that leads to knowledge development. The interaction with OER course content 
demonstrates the knowledge and explains the essential concepts about the course through the 
entire semester. It was found that the learners preferred to view through the hyperlinks OER 
resources (13%) as compared to the E-Books and slides (10%). The viewing activity was usually 
influenced by the content of the WawasanLearn and the posting of message in the forum. In simple, 
the good use of the OER resources as additional supports in the WawasanLearn was able to attract 
students to view the LMS and was able to support the students’ learning. 

Key Findings: Continuous Assessments and Examination Score 

Students performed satisfactory in the OCAS (Overall Continuous Assessment Score) during 
January 2013 (OER) with a mean score of 75.14% and a standard deviation (S.D) of 8.82 as shown 
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□ OER Course Content (Course Materials) 

■ Hyperlinks to OER Resources and 
Additional Resources 

□ Online Discussion Forums 

□ E-Books and Power Point Slides 


Figure 5: Analysis from the activity reports of the learners’ use of online resources in 

WawasanLearn 


in Table 4 below. On the other hand, learners in July 2012 scored lower OCAS with a mean score 
of 68.84% and S.D 7.10. This reflects the learners tend to perform better in the January 2013 
semester than July 2012 semester as the additional OER resources are perceived as a valuable 
learning resource to test learners’ understanding and allow learners to be better prepared for the 
TMA and final examination. 

The assessments of TCC121/05 comprises Tutor Marked Assignments (TMA), mini projects and 
problem-solving case studies which were given by the Course Coordinator and tutor in the LMS for 
more in depth Java learning. Mini projects in the TMA tested the students’ programming skills, 
problem solving and analyzing skills, flow chart/diagram and apply major components in Java 
programming in producing solution design. 

As observed in Table 5, the Overall Examination Score (OES) achieved is 54.18%, with a 
standard deviation of 12.56. The OER-based student performance on the OES is considered to be 
above average as compared with non OER TCC121/05 with a mean score of 50.98% and standard 
deviation 11.17. 


Table 4: Comparison of Overall Continuous Assessment Score 


Overall Continuous Assessment Score 

January 2013 (OER) 

July 2012 (Non OER) 

Mean 

75.14 

68.84 

Standard Deviation 

8.82 

7.10 


Table 5: Comparison of Overall Examination Score 


Overall Examination Score 

January 2013 (OER) 

July 2012 (Non OER) 

Mean 

54.18 

50.98 

Standard Deviation 

12.56 

11.17 
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Table 6: OCAS, OES and Course Score For Programming Fundamentals with Java 



OCAS (50%) 

OES (50%) 

Course Score (100%) 


Jan 2013 
(OER) 

July 2012 
(Non OER) 

Jan 2013 
(OER) 

July 2012 
(Non OER) 

Jan 2013 
(OER) 

July 2012 
(Non OER) 

Mean 

37.57 

34.42 

27.09 

25.49 

64.66 

59.91 

S.D 

8.82 

7.70 

12.56 

11.17 

21.38 

18.87 


The summary of the students’ overall performance is illustrated in Table 6 above. The mastery 
of the TCC121/05 course is evaluated via an assessment strategy that consists of assignments, 
OCAS weighted at 50% and a proctored final examination, OES weighted at 50% to be computed 
as overall Course Score, 100%. Relatively, the students scored better marks in the January 2013 
semester as they can obtain supports from the additional OER resources in the completion of their 
TMAs. This has reflected that the learners of OER-based TCC121/05 have shown better results 
and achievements in overall assessment and examination score respectively. 

Key Findings: Student Survey on Teaching and Learning Effectiveness 

Table 7 summarizes the relevant statistics scored for the seven items in the student survey of 
TCC121/05 teaching and learning effectiveness conducted in January 2013 semester (OER). The 
pertinent data are the average (maximum of 5.00) of each aspect and comparison is done across 
courses in WOU. There were 95 students out of 124 students participated in the survey. The 
students were satisfied with the tutorials, course materials, TMAs, library, MyDigital Library and also 
supports provided via WawasanLearn. 

Recommendation and Conclusion 

This paper presents the development and evaluation of OER resources in WOU learning materials 
for delivery in ODL mode. The course development cycle includes the interaction among students 
and tutors with feedbacks sessions and learning experiences of courses units’ content as one of 
the way in mobilising the faculty to support the use and re-use of OER. The course team members 
recommend that subsequent revision will be carried out on the TCC121/05 OER-based course 


Table 7: Students Survey (95 respondents) 



Item Descriptions 

Average Score 

Overall 

1 

Face to face tutorials 

3.99 

4.01 

2 

Course materials 

3.81 

3.53 

3 

Learning Management System ( WawasanLearn ) 

3.90 

3.63 

4 

Library 

3.51 

3.01 

5 

MyDigital Library 

3.54 

2.92 

6 

Tutorial Assistance through Telephone, Email and WawasanLearn 

3.86 

3.67 

7 

Tutor Marked Assignments 

3.92 

3.82 
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units subjected to changes in technology impacting the course. The findings have indicated that 
the development framework will speed up the creation of course materials (spans approximately 
6 months) and eliminate the necessity of accompanying textbook to achieve long-term cost- 
effectiveness in educational practice. The implementation strategy used in this study is targeted 
to promote and encourage collaboration in the creation and integration of OER courses within an 
Open Licensing framework. The above findings have also shown that TCC121/05 OER-based 
course learners are adapting well to the open course in online distance learning environment. In 
fact, the learners have shown great participation in, and utilization of, the open course content and 
OER online resources in the LMS. This initiative have also provided opportunities for ODL educa¬ 
tors to share experience and encourage further learning of OER integration via virtual workshops 
in widening the participation of OER learning environment and spreading knowledge through ODL 
environment. 

Acknowledgments 

This paper has been awarded an ICDE Prize for Innovation and Best Practice on the theme of Open 
Educational Resources and intercultural aspects at the 25 th ICDE Conference, held in Tianjin (China) 
in October 16 th -18 th 2013. 

References 

Atkins, D. E., Brown, J. S. & Hammond, A. L. (2007). A Review of the Open Educational Resources 
(OER) Movement: Achievements, Challenges, and New Opportunities. Menlo Park, CA: The 
William and Flora Hewlett Foundation, 4. 

Chung, S. H. & Khor, E. T. (2012). Framework for Development of OER-based Learning Materials. 
Proceedings of the Regional Symposium on Open Educational Resources: An Asian perspective 
on policy and practices, 20-24. 

Hilton III, J., Wiley, D., Stein, J. & Johnson, A. (2010). The four R’s of openness and ALMS 
analysis: frameworks for open educational resources. Open Learning: The Journal of Open and 
Distance Learning, 25(1), 37-44. http://dx.doi.org/10.1080/02680510903482132 
Jonassen, D. H. (1995). Constructivism and Computer-mediated Communication in Distance 
Education. American Journal of Distance Education, 9(2), 7-25. 

McGreal, R., Kinuthia, W., Marshall, S. & McNamara, T. (eds.). (2013). Perspectives on Open and 
Distance Learning: Open Educational Resources: Innovation, Research and Practice. Canada: 
Commonwealth of Learning, Athabasca University. Retrieved from http://www.col.org/resources/ 
publications/Pages/detail.aspx?PID=446 

MELT. (2010). Metadata Ecology for Learning and Teaching. European Commission’s eContentplus 
Programme. 

Menon, M. & AN, P. K. M. (2012). Learning from advocacy: A case of OER policy implementation in 
the Wawasan Open University. Proceedings of the Regional Symposium on Open Educational 
Resources: An Asian perspective on policy and practices, 1-5. 

OECD. (2007). Giving Knowledge for Free, The Emergence of Open Educational Resources. 

Organisation for Economic Co-operation and Development (OECD) Publishing. 

SOP for Course Development. (2010). Standard Operating Procedure (SOP) for Course 
Development. Wawasan Open University, Penang, Malaysia. 

Yuan, L. & Powell, S. (2010). MOOCs and Open Education: Implications for Higher Education. JIST 
CETIS. 


Papers are licensed under a Creative Commons Attribution 3.0 Unported License 


Open Praxis, vol. 5 issue 4, October-December 2013, pp. 315-324 


